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FOREWORD 


This  study  was  performed  under  Work  Unit  Number  ZPN07.30  (Adaptive 
Experimental  Approach  to  Instructional  Design^" while’  LIP.  KulTTavy  of 
Arizona  State  University  was  a temporary  employee  of  NAVPERSRANDCEN 
during  the  summer  of  1975.  It  falls  within  the  Instructional  Design 
Support  task  area,  which  focuses  on  theories  and  procedures  useful  in 
supporting  instructional  development  activities  within  the  Navy. 

Dr.  John  Carter  served  as  project  director  for  this  work. 


J.  J.  CLARKIN 
Commanding  Officer 


SUMMARY 


Problem 


Various  feedback  procedures  are  used  in  Navy  training  programs  to 
facilitate  learning  processes.  Unfortunately,  the  mechanisms  respon- 
sible for  such  facilitation  are  frequently  misunderstood.  As  a result, 
feedback  is  often  used  inappropriately,  thereby  neutralizing  any  positive 
effects  it  might  have  on  student  performance. 


Purpose 

To  review  the  data  available  on  feedback  processes  and  to  show  how 
post-response  information  can  best  be  utilised  with  written  instruction 


Approach 


A review  was  made  of  studies  conducted  on  the  effect  of  providing 
feedback  with  text-based  lessons.  Results  were  organized  under  five  topic 
headings:  Feedback  Complexity,  Feedback  as  Reinforcement,  Effect  of  High 

Feedback  Availability,  Feedback  and  Learning,  and  Feedback  and  Learner 
Confidence. 


Findings 


1.  The  literature  review  provided  no  decisive  evidence  that  increases 
in  feedback  complexity  yield  parallel  increases  in  learning  or  that  feed- 
back necessarily  functions  as  reinforcement. 


2.  High  feedback  availability  negates  its  facilitation  effect,  since 
students  merely  copy  answers  and  learn  little. 

3.  If  feedback  availability  is  controlled  and  the  learner  is  familiar 
with  subject  matter,  feedback  serves  (a)  to  let  the  student  know  when  he 
is  right  or  (b)  to  correct  him  (or  let  him  correct  himself)  if  he  is  wrong 


4.  Feedback  is  most  effective  when  student  response  confidence  is  high, 
whether  or  not  his  initial  response  is  correct. 


Conclusions 


Feedback  facilitation  is  negated  if  (1)  feedback  availability  is  high 
and  he  merely  copies  instead  of  reading  the  material  or  (2)  the  material 
studied  is  so  difficult  that  the  student  must  guess  at  answers  and  then 
try  to  associate  feedback  with  the  question;  otherwise  feedback  acts  to 
confirm  correct  responses  or  provides  corrective  action. 


Rscoenendatioas 


To  ensure  that  feedback  provide#  narlsun  learning  facilitation,  it 
la  reconnended  that  lnatructlonal  designer#: 

1.  Ensure  that  learners  have  sufficient  entry  skills  for  the  lesson. 

2.  Minimize  the  opportunity  for  students  to  obtain  feedback  prior  to 
responding. 

3.  Provide  feedback  as  often  as  possible  during  the  course  of  the 
leeson. 

It  is  further  recoasended  that  future  research  in  this  area  be  directed 
toward  the  relationship  between  learner  confidence  end  feedback. 
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INTRODUCTION 


Various  feedback  procedures  are  used  in  Navy  training  programs  to 
facilitate  learning  processes.  Unfortunately,  the  mechanisms  responsible 
for  such  facilitation  are  frequently  misunderstood.  As  a result,  in 
both  research  and  instructional  development,  feedback  is  often  used  in- 
appropriately, thus  neutralizing  any  positive  effects  it  might  have  on 
student  performance. 

The  main  purpose  of  this  report  is  to  review,  analyze,  and  interpret 
results  of  studies  performed  on  feedback  processes,  especially  as  they 
apply  to  written  instruction  and  the  design  of  instructional  materials.--^ 

No  attempt  was  made  to  cover  the  more  task-specific  literature  dealing  ' \v 
with  motor  learning,  simple  concept  acquisition,  or  paired-associates, 
since  excellent  reviews  already  exist  in  these  areas  (Adams,  1968,  Bilodeau, 
1966;  Bourne,  1966;  Renner,  1964). 

APPROACH 


A review  of  studies  on  the  effect  on  providing  feedback  with  text-based 
lessons  was  conducted.  Results  were  organized  under  five  topic  headings: 
Feedback  Complexity,  Feedback  as  Reinforcement,  Effect  of  High  Feedback 
Availability,  Feedback  and  Learning,  and  Feedback  and  Learner  Confidence. 
The  section  on  Feedback  and  Learning  includes  information  processing 
approaches  and  a possible  application  to  automated  teaching  systems. 


FINDINGS 


Feedback  Complexity 


Throughout  this  report  the  term  "feedback"  is  used  to  describe  any 
of  the  numerous  procedures  used  to  tell  a learner  whether  his  response 
is  right  or  wrong.  As  shown  by  Figure  1,  feedback  can  range  from  the 
simplest  "Yes-No"  format  to  the  presentation  of  substantial  corrective 
or  remedial  information  which  may  extend  or  add  to  the  response  content. 
Obviously,  as  one  advances  along  this  range,  feedback  complexity  increases 
until  the  process  takes  on  the  form  of  new  instruction  rather  than  informa- 
tion to  the  student  about  response  correctness. 


Common  sense  suggests  that  increments  in  the  complexity  of  feedback 
should  increase  learning.  Intuitively,  the  more  information  a student 
has  about  his  response,  the  better  he  should  understand  why  he  made  it. 
Unfortunately,  the  literature  provides  no  decisive  evidence  that  increases 
in  complexity  yield  parallel  increases  in  learning.  Rather,  research  on 
this  point  yields  inconsistent  results.  For  example,  studies  comparing 
the  relatively  simple  feedback  in  linear  programs  to  more  extensive  format 
often  found  no  posttest  performance  differences  (Bisbiscos,  1965;  McDonald 
and  Allen,  1962;  Merrill,  1965;  Roe,  1962).  Others  found  that  adding 
redundant  content  to  a confirmation  does  little  more  than  create  extra 
work  for  the  writer  (Kaufman,  1963;  O'Day,  Kulhavy,  Anderson,  and  Malczynski 
1971).  On  the  other  hand,  it  is  sometimes  possible  to  increase  performance 
by  making  the  feedback  so  extensive  that  it  constitutes  a partial  review 
of  the  lesson  (Block  and  Tierney,  1974;  Merril  and  Stolurow,  1966)  or  by 
simultaneously  giving  several  types  of  feedback  following  each  response 
(Gilman,  1969).  Clearly,  these  studies  allow  no  systematic  statement 
about  how  changes  in  feedback  form  or  content  are  likely  to  influence 
achievement.  Such  a statement  may  be  impossible  to  make  because  of  the 
numerous  ways  in  which  individual  differences  influence  learning  from  a 
specific  block  of  material.  Consequently,  until  more  definitive  research 
becomes  available,  it  is  probably  better  to  treat  feedback  as  a unitary 
concept,  leaving  the  complexity  issue  unresolved. 


Restatement  and  Remediation 


COMPLEXITY 


" I 

Restatement,  Remediation, 
and  New  Information  about 
Response 


Figure  1.  Types  of  feedback. 
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Feedback  as  Reinforcement 


It  has  often  been  proposed  that  feedback  acts  as  a reinforcing  event. 
This  belief  goes  back  to  the  early  fifties,  and  the  development  of  pro- 
grammed instruction  materials  for  classroom  use.  The  principle  of  immediate 
feedback  found  its  strongest  support  among  operant  psychologists  who  argued 
that  telling  a pupil  his  answer  is  right  "reinforces"  him  and  increases 
the  likelihood  that  he  will  again  respond  correctly  on  a future  test. 

In  fact,  as  late  as  1968,  ' .inner  himself  concluded  a discussion  on 
teaching  machines  with  the  >llowing  observation:  "the  machine,  like 

any  private  tutor,  reinforces  the  student  for  every  correct  response, 
using  the  immediate  feedback  not  only  to  shape  behavior  most  effectively 
but  to  maintain  it  in  strength  in  a manner  which  the  layman  would  describe 
as  'holding  the  student's  interest'"  (p.  39).  With  such  positive  statements 
available,  it  is  no  surprise  that  scholars  have  worked  overtime  to  fit 
the  round  peg  of  feedback  into  the  square  hole  of  reinforcement.  Unfor- 
tunately, this  stoic  faith  in  feedback-as-reinforcement  has  all  too  often 
lead  researchers  to  overlook  or  disregard  alternate  explanations  when  their 
data  failed  to  meet  operant  expectations. 

The  problem,  of  course,  is  that  behaviorists  were  a bit  hasty  in 
assigning  functionally  reinforcing  properties  to  feedback.  In  the  labora- 
tory, a reinforcer  is  defined  as  some  stimulus  which  increases  the  prob- 
ability that  the  response  which  it  follows  will  occur  again.  For  example, 
suppose  that  a laboratory  rat  or  pigeon  is  deprived  of  food  and  then  is 
required  to  perform  some  target  response  in  order  to  be  fed.  When  the 
animal  responds  correctly,  he  receives  a food  pellet.  To  the  degree  that 
he  will  emit  the  same  response  again  under  similar  conditions,  the  pellet 
assumes  reinforcing  properties.  There  is  no  good  reason  to  believe  that 
this  same  sequence  of  events  will  occur  in  programmed  text.  Obviously, 
classroom  learners  rarely  perform  under  such  contingencies  as  physical 
depreviation.  Further,  the  stimulus-response  environment  of  the  program 
is  constantly  undergoing  change,  rather  than  being  controlled  in  the  typical 
laboratory  setting.  This  does  not  mean  that  some  program  stimuli  are  not 
functionally  reinforcing  for  some  students.  However,  to  assume  that  the 
instructional  behavior  of  most  pupils  is  subject  to  the  control  of  feed- 
back statements  is  to  violate  the  operant  canon  itself:  only  that  which 

reinforces  is  a reinforcer. 

The  author  has  been  unable  to  find  evidence  that  the  feedback  which 
follows  program  frames  typically  acts  in  a reinforcing  manner.  However, 
despite  repeated  challenges  to  the  operant  position  over  the  years 
(Anderson,  1967,  1970;  Annett,  1969;  Smith  and  Smith,  1966),  it  still 
has  considerable  currency  in  instructional  design  and  research.  This 
state  of  affairs  seems  doubly  strange  in  light  of  the  vast  amount  of 
feedback  data  which  prove  difficult  to  handle  within  a reinforcement 
framework.  For  instance,  a study  by  Anderson,  Kulhavy,  and  Andre  (1971, 

Kxp.  1)  required  two  groups  of  college  students  to  learn  a 112-frame, 


computer-controlled  lesson  about  heart  disease.  In  one  group,  subjects 
were  provided  feedback  only  after  correct  responses,  and  in  another  only 
after  Incorrect  responses.  Both  groups  made  about  the  same  number  of 
errors  during  learning,  and  there  were  no  significant  differences  on  the 
posttest.  Actually,  the  wrong-only  learners  made  higher  test  scores, 
in  spite  of  the  fact  that  they  received  less  than  one-half  the  amount 
of  feedback  given  the  right-only  group.  These  results  clearly  show  that 
feedback  does  not  have  its  greatest  effect  on  correct  responses,  a condition 
implicit  in  the  reinforcement  position. 

Delay-Retention  Effect  Studies 


Research  dealing  with  the  so-called  Delay-Retention  Effect  (DRE) 
also  presents  difficulty  for  a reinforcement  interpretation.  Studies 
on  the  DRE  have  repeatedly  shown  that  delaying  the  presentation  of  feed- 
back for  a day  or  more  significantly  increases  what  students  remember  on  a 
retention  test  (Moore,  1969;  Sassenrath  and  Yonge,  1968,  1969;  Sturges, 
1969,  1972).  These  consistent  results  sharply  oppose  those  that  would  be 
expected  if  feedback  acts  as  a reinforcer,  since  one  of  the  surest  ways  to 
destroy  whatever  control  a reinforcer  has  over  behavior  is  to  separate  it 
from  the  response  by  a lengthy  interval  (cf.  Renner,  1964).  In  fact,  in 
at  least  two  DRE  studies  (Kulhavy  and  Anderson,  1972;  Surber  and  Anderson, 
1975)  it  was  shown  that  people  remember  correct  answers  as  well  when  feed- 
back is  delayed  for  a day  as  when  it  is  given  immediately  after  the  re- 
sponse. Kulhavy  and  Anderson  summarize  the  argument  when  they  say,  "The 
data  from  this  study  fail  to  support  the  view  that  feedback  acts  as  re- 
inforcement. If  feedback  were  reinforcing,  immediate  feedback  would  be 
expected  to  increase  the  likelihood  of  repeating  initially  correct  re- 
sponses. In  fact,  the  probability  of  repeating  correct  responses  on  the 
final  (delay)  test  was  no  higher  for  immediate-feedback  groups  than  for 
delay-feedback  groups." 

Feedback  Schedule  Studies 

Earlier  it  was  contended  that  a preoccupation  with  reinforcement 
principles  has  forced  many  studies  into  nonproductive  designs.  Nowhere  is 
this  phenomenon  more  apparent  than  with  research  attempting  to  replicate 
the  effects  of  reinforcement  "schedules"  with  programmed  texts  (Krumboltz 
and  Weisman,  1962;  Lublin,  1965;  Rosenstock,  Moore,  and  Smith,  1965). 
Typically,  subjects  in  these  experiments  receive  feedback  on  only  a per- 
centage of  the  frames  studied,  and  their  test  performance  is  then  compared 
with  that  of  learners  who  see  feedback  after  every  frame  or  not  at  all. 

The  number  of  frames  after  which  feedback  is  given  can  range  from  10  to 
67  percent,  and  the  feedback  can  occur  either  following  a specified  number 
of  responses  (fixed  ratio)  or  randomly  as  a percentage  of  the  total  program 
(variable  ratio).  In  spite  of  the  breadth  of  subject  matters  and  feed- 
back percentages  used  in  this  research,  not  one  study  showed  a pattern 
of  results  consistent  with  what  one  might  expect  from  work  done  with 
intermittent  reinforcement.  In  fact,  in  discussing  these  experiments, 
Anderson  (1967)  pointed  out  that  it  is  unclear  what  studies  on  feedback 
schedules  hope  to  accomplish,  since  the  lower  ratios  are  at  a clear  dis- 
advantage in  the  first  place.  However,  so  powerful  was  the  Zeitgeist 


6 


that  one  article  declared  that,  "the  most  prominent  reinforcer  is  the  know- 
lege  of  results  provided  by  the  confirming  answer.  Providing  confirming 
answers  on  an  intermittent  schedule  during  learning  of  programmed  material 
may  increase  retention  in  the  same  manner  that  intermittent  reinforcement 
prolongs  responding  in  less  complex  learning  tasks"  (Krumboltz  and  Weisman, 
1962,  p.  250).  Since  data  have  failed  to  support  these  strong  expecta- 
tions, it  seems  reasonable  to  conclude  that  such  studies  not  only  provide 
a weak  test  of  the  reinforcement  position  but  also  appear  to  add  little 
to  what  we  know  about  how  feedback  works  in  an  instructional  environment. 

Incentive  Studies 

A final  approach  to  the  reinforcement  question  comes  from  studies 
which  have  combined  some  "extrinsic  reinforcer"  with  feedback  during  the 
instructional  sequence.  Such  research  is  not  really  concerned  with  how 
feedback  itself  influences  learning  but,  rather,  with  how  some  tangible 
incentive,  external  to  the  material,  interacts  with  various  lesson  designs. 
The  traditional  argument  has  been  that  providing  extra-instructional  re- 
wards for  good  performance  will  lead  to  increases  in  test  scores,  simply 
because  the  reward  causes  students  to  take  maximum  advantage  of  posi- 
tive variables  such  as  feedback.  Predictably,  incentive  studies  first 
attempted  to  control  study  behavior  by  giving  the  subjects  money  for 
producing  at  a high  level.  Moore  and  Smith  (1964)  gave  undergraduates 
one  penny  for  each  correct  response  to  a prepublication  version  of  the 
Analysis  of  Behavior  (Holland  and  Skinner,  1961),  but  the  money  had  no 
significant  effect  on  either  multiple-choice  or  constructed  response 
test  scores.  In  a later  study,  Sullivan,  Baker,  and  Schutz  (1967)  pro- 
vided subjects  with  either  $2.50  just  for  attending  the  experiment  or 
$4.50  for  completing  the  criterion  test  with  a high  score.  Again,  there 
were  no  posttest  differences,  nor  did  the  performance  of  incentive  groups 
vary  as  a function  of  whether  feedback  was  presented  during  the  lesson. 
These  same  authors  (Sullivan,  Schutz,  and  Baker,  1971)  conducted  a second 
study  in  which  they  allowed  students  to  bypass  ROTC  drill  periods  in 
return  for  high  performance  with  a lesson  on  The  Military  Justice  System. 
The  subjects  responded  to  short  mastery  tests  during  instruction  and  also 
received  a cumulative  final  over  the  content.  Although  performance  re- 
sulting from  seeing  the  feedback  immediately  following  mastery  tests  was 
superior  to  that  resulting  from  delaying  it  a session,  there  were  no 
reliable  effects  for  the  incentive  variables  nor  did  it  interact  with 
feedback  placement.  Unfortunately,  this  experiment  contained  no  group 
where  feedback  was  not  given,  so  it  is  impossible  to  determine  if  the 
incentive  influenced  how  effectively  subjects  used  their  postresponse 
information.  Results  from  these  three  studies  supply  no  evidence  that  in- 
centives external  to  the  lesson  lead  to  better  performance,  at  least  with 
the  type  of  materials  used  in  this  research. 

In  conclusion,  it  is  difficult  to  find  data  indicating  that  feed- 
back following  written  instruction  functions  in  the  manner  predicted  by 
Skinner  and  others.  One  is  forced  to  accept  the  fact  that,  whatever  feed- 
back does,  it  rarely  acts  as  a reinforcer  with  text-based  materials. 
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Effect  of  High  Feedback  Availabilit 


A number  of  studies  have  found  that  feedback  has  no  effect  on  learning 
(Keldhusen  and  Brit,  1962;  Hough  and  Revsin,  1963;  Krumboltz  and  Weisraan, 
1962;  McDonald  and  Allen,  1962;  Moore  and  Smith,  1961,  1964;  Rosenstock, 
Moore,  and  Smith,  1965;  Sullivan,  Baker,  and  Schutz,  1967;  Sullivan, 

Schutz,  and  Baker,  1971;  Wentling,  1973).  Some  research  even  shows  that 
groups  who  received  no  feedback  or  who  never  saw  the  questions  during 
learning  performed  as  well  as,  or  better  than,  groups  who  saw  feedback 
after  every  response  (Karraker,  1967;  Lublin,  1965).  Such  results  are 
counterintuitive,  and  difficult  to  explain  until  one  looks  closely  at 
the  materials  and  procedures  used  in  many  of  these  experiments. 

With  minor  exceptions,  researchers  who  found  no  effect  for  feedback 
used  text  format  which  either  allowed  learners  to  see  the  feedback  before 
responding  or  which  were  so  heavily  cued  and  prompted  that  students  could 
answer  questions  correctly  after  only  a cursory  reading.  It  is  not  sur- 
prising, then,  that  even  though  there  was  no  effect  on  learning,  some  of 
these  studies  found  that  the  feedback  groups  make  consistently  fewer  errors 
than  their  no-feedback  counterparts  (Krumboltz  and  Weisman,  1962;  Moore 
and  Smith,  1964;  Rosenstock,  Moore,  and  Smith,  1965);  and  others,  that  they 
took  far  less  time  to  complete  the  instruction  (Lublin,  1965;  Moore  and 
Smith,  1964) . One  possible  reason  why  the  feedback  students  in  these 
studies  made  few  errors  and  took  minimum  time — but  learned  little — is  that 
they  merely  copied  answers  instead  of  reading  test  (cf.  Anderson,  1970). 
Alternately,  subjects  receiving  no  feedback  could  not  access  the  correct 
answers  and  thus  were  forced  to  read  the  material  in  order  to  make  a 
feasible  response.  Under  these  conditions,  the  mere  fact  that  no-feedback 
students  studied  the  text  is  sufficient  to  overcome  any  advantage  that  the 
feedback  students  might  have  had  on  a subsequent  posttest. 

This  argument  is  supported  by  the  fact  that  three  of  the  five  studies 
which  showed  significant  learning  increases  for  feedback  were  conducted 
using  computer-controlled  lessons  where  feedback  was  impossible  to  obtain 
until  after  the  learner  had  typed  his  response  into  the  computer  (Anderson 
et  al.,  1971,  1972;  Gilman,  1969).  The  fourth  used  a punchboard  procedure 
where  feedback  was  unavailable  until  a response  was  actually  made  (Angell, 
1949).  In  the  final  experiment,  Meyer  (1960)  actively  discouraged  looking 
ahead  at  answers.  In  fact,  he  discarded  answers  from  four  students  in  the 
experiment  because  they  were  caught  looking  ahead  at  the  answers.  The 
conditions  in  Meyer's  study  were  quite  different  from  those  of  many  experi- 
ments, where  the  correct  answer  is  printed  directly  below  the  question 
or  on  the  next  page  within  easy  reach. 

When  "answer  availability"  (the  ease  with  which  a learner  can  locate 
correct  answers  without  first  reading  the  lesson)  is  high,  students  simply 
copy  the  responses  and  bypass  most  of  the  instruction--a  condition  known  to 
yield  little  learning  (Anderson  and  Faust,  1967).  On  the  other  hand. 
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when  answer  availability  is  low,  students  must  study  the  material  in 
order  to  respond  properly.  The  sequence  which  relates  answer  availability 
and  lesson  behavior  is  diagrammed  in  Figure  2.  The  availability  hypothe- 
sis has  been  directly  tested  in  the  Anderson,  et  al.  studies  (1971,  Exp.  2; 
1972)  mentioned  above.  As  noted,  these  studies  presented  college  students 
with  various  feedback  arrangements  using  computer-controlled  lessons, 
both  experiments  included  a condition  labeled  PEEK,  in  which  the  correct 
answer  to  each  program  frame  was  visible  in  the  lower  corner  of  the  computer 
screen  during  the  entire  time  that  the  item  was  exposed.  Subjects  in  the 
PEEK  groups  were  repeatedly  admonished  not  to  look  at  the  feedback  before 
constructing  their  response.  However,  as  the  succession  in  Figure  2 
predicts,  students  in  the  PEEK  groups  completed  the  lesson  in  record  time, 
made  few  program  errors,  and  learned  very  little  from  the  experience. 

In  each  experiment,  subjects  who  see  feedback  after  every  frame,  with 
answer  availability  controlled,  had  significantly  higher  scores  on  an 
immediate  posttest.  The  effects  of  high  answer  availability  are  apparent 
in  the  comments  made  by  PEEK  subjects  on  a postexperimental  questionnaire. 
For  example,  one  student  wrote,  "The  answers  should  not  be  shown  immedi- 
ately, as  this  causes  cheating  to  be  easier"  (Anderson,  et  al. , 1971,  Exp. 
2),  and  another,  "It  was  difficult  for  me  not  to  use  the  answer  when  the 
question  was  difficult  for  me"  (Anderson,  et  al.,  1972).  Obviously,  even 
when  students  are  told  not  be  peek  at  a highly  available  answer,  they  are 
likely  to  copy  anyway,  rather  than  puzzle  through  a difficult  frame. 

The  evidence  concerning  answer  availability  leads  to  two  firm  con- 
clusions. First,  availability  must  be  low  if  one  hopes  to  perform  valid 
comparisons  among  different  feedback  arrangements.  If  answers  are  easily 
accessible,  students  will  merely  copy  them,  preventing  a valid  comparison 
between  various  feedback  conditions.  Second,  from  an  instructional  point 
of  view,  the  designers  of  teaching  materials;  should  control  answer  avail- 
ability so  that  feedback  works  primarily  to  teach  content  and  not  copying. 

Feedback  and  Learning 

The  preceding  discussion  outlines  some  of  the  reasons  why  many  studies 
with  feedback  have  failed  to  produce  a decisive  result.  This  section 
discusses  how  feedback  facilitates  learning  when  such  factors  as  answer 
availability  are  controlled  (e.g.,  Anderson,  et  al. , 1971;  Gilman,  1969; 
Meyer,  1960).  One  can  approach  this  task  by  evoking  concepts  used  in  the 
study  of  cybernetics,  especially  the  area  of  servocontrol  systems.  Within 
this  context,  feedback  functions  as  a means  for  (1)  acquiring  data  about 
how  accurately  a system  is  working  and  (2)  identifying  and  correcting 
error  messages  (Smith  and  Smith,  1966).  If  this  is  applied  to  written 
instruction,  feedback  will  either  let  the  student  know  when  he  is  right, 
or  correct  him  (or  let  him  correct  himself)  when  he  is  wrong.  The  argu- 
ment is  that  this  continuous  confirmation-correction  process  is  responsible 
for  increases  in  learning  which  occur  when  feedback  is  present. 
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Figure  2.  Effect  of  high  answer  availability. 


10 


Within  the  configuration  of  information  processing,  the  student  and 
the  lesson  are  viewed  as  components  of  a self-regulating  control  system 
which  has  the  terminal  goal  of  transferring  information  from  the  text 
to  the  learner  (Talyzina,  1973).  This  system  assumes  that  the  student 
possesses,  and  is  capable  of  using,  at  least  some  prior  knowledge  related 
to  the  material  being  studied.  This  assumption  seems  fair,  since  it 
is  a rare  case  indeed  when  learners  know  nothing  about  the  subject  matter 
on  which  they  are  concentrating.  Under  these  conditions,  the  response 
to  each  question  is  based  on  both  general  knowledge  related  to  the  item 
and  on  specific  information  covered  within  the  text  just  read.  In  other 
words,  the  feedback  acts  to  inform  the  student  how  accurate  his  response 
is  relative  to  the  knowledge  he  already  possesses.  Obviously,  if  the 
material  being  studied  is  abstruse  or  unfamiliar  to  the  student,  providing 
feedback  should  have  little  effect  on  his  criterion  performance,  since  he 
has  no  way  to  relate  the  new  information  to  what  he  already  knows. 

Feedback  Confirming  a Correct  Response 

When  feedback  follows  a correct  response,  it  tells  the  learner 
that  his  understanding  is  adequate  to  that  point  in  the  material.  Selec- 
tion of  the  right  answer  produces  a "match"  with  the  feedback,  and  the 
student  knows  that  he  interpreted  the  question  correctly.  Confirming 
feedback  provides  a mechanism  within  the  student-lesson  system  for  in- 
forming the  controlling  unit  (student)  that  the  comprehension  strategy 
is  achieving  the  terminal  goal  of  transferring  the  information  from  text 
to  learner.  This  sequence  of  events  satisfies  the  requirement  that  the 
system  has  a means  for  furnishing  data  about  goal  accuracy. 

The  comprehension  explanation  raises  the  question  of  whether  or 
not  feedback  has  a direct  effect  on  individual  items  which  are  answered 
correctly.  As  we  shall  see  later,  there  are  cases  where  correct  test 
responses  appear  to  be  dependent  on  performance  during  learning.  However, 
both  correct  and  incorrect  answers  have  a remarkable  tendency  to  persever- 
ate  to  later  tests,  regardless  of  whether  or  not  feedback  is  available 
(Kaess  and  Zeaman,  1960;  Kulhavy  and  Parsons,  1972;  Kulhavy  and  Anderson, 
1972).  Consequently,  since  right  answers  have  a proclivity  for  repeating 
themselves,  any  positive  effects  due  to  confirmation  are  probably  geared 
more  to  overall  comprehension  than  to  the  clarification  of  specific  items. 
This  interpretation  assumes,  of  course,  that  the  initial  responses  are 
based  on  prior  knowledge,  rather  than  a random  guess. 


Feedback  Correcting  an  Incorrect  Response 


In  terms  of  increased  learning,  supplying  corrective  feedback  after 
an  incorrect  answer  is  probably  far  more  important  than  providing  confirma- 
tion after  a correct  one.  In  the  case  of  a correct  answer,  no  change  is 
required.  However,  when  an  incorrect  answer  is  produced,  the  object  is 
not  only  to  eliminate  the  wrong  answer  but  also  to  substitute  the  correct 
one.  This  prescription  leads  to  the  seasoned  notion  that  feedback  on 
errors  acts  primarily  as  a corrective  agent  (cf.  Anderson,  1967,  1970; 
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Annett,  1969).  Here  feedback  fulfills  the  requirement  that  the  processing 
system  has  some  way  of  identifying  error  responses.  Once  the  error  Is 
recognized,  the  student  can  attempt  to  eliminate  It  and  replace  It  with 
correct  information. 


One  reason  why  the  corrective  aspects  of  feedback  have  received 
so  little  attention  is  simply  because  many  studies  fail  to  analyze  In- 
correct and  correct  responses  separately.  However,  when  the  experimental 
treatments  themselves  emphasize  error  responding,  the  data  consistently 
indicate  a corrective  effect.  For  example,  in  a series  of  studies  on 
concept  acquisition,  Buss  and  his  associates  repeatedly  found  that  telling 
a subject  when  he  was  wrong  yielded  higher  criterion  performance  than  con- 
firming correct  answers  (Buss,  Braden,  Orgel,  and  Buss,  1956;  Buss  and  Buss, 
1956;  Buss,  Weiner,  and  Buss,  1954).  The  corrective  effect  of  feedback 
is  also  apparent  in  a study  by  Travers,  Van  Wagenen,  Haygood,  and  McCormick 
(1964)  who  used  various  feedback  combinations  to  teach  school  children 
German  vocabulary  words.  Children  who  were  told  that  a response  was 
wrong  and  then  were  corrected  did  far  better  than  those  who  received  a 
simple  yes  or  no  following  each  response.  Finally,  in  the  Anderson, 
et  al.  (1971)  experiment  described  earlier,  posttest  scores  for  groups 
who  saw  corrective  feedback  only  after  they  had  given  an  incorrect  response 
or  who  were  allowed  extra  study  time  to  determine  why  they  had  made  wrong 
responses  were  not  appreciably  different  from  those  for  the  group  who  saw 
feedback  following  each  frame.  This  research  suggests  that  feedback  works 
to  correct  wrong  responses. 


An  even  more  decisive  argument,  however,  comes  from  work  done 
on  the  Delay-Retention  Effect  (DRE) . As  mentioned  earlier,  studies  em- 
ploying the  DRE  consistently  show  that  when  feedback  is  delayed  for  some 
period  following  learning  rather  than  being  presented  immediately,  signi- 
ficant increases  in  learning  occur.  DRE  experiments  have  five  components 
which  can  be  represented  schematically  as: 


FB 


Where: 


R^  signifies  initial  exposure  to  the  stimuli  to  be  learned, 

generally  involving  a response  selection  on  the  part  of  the 
subj ect; 


d^,  a delay  interval  of  some  time  t^; 

FB,  the  Rj^  l 
(feedback) ; 


FB,  the  R^  test  responses  with  the  correct  answer  identified 


r.,  a retention  interval  of  some  time  t ; 
i r 


and 


R^»  the  second  presentation  of  the  test  stimuli  (dependent 


measure) . 
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The  typical  DRE  design  includes  two  groups  who  both  receive  R^, 

FB,  and  R^.  As  shown  below,  the  delay  group  sees  feedback  following  some 

specified  interval  after  R,  (t,  = c) ; and  the  no-delay  group,  immediately 

1 d 

following  R^  (t^  = 0).  For  both  groups  t^  + t = n. 

DELAY  GROUP 

R.  FB R 

(td  = c)  (tr  = n-c) 

NO-DELAY  GROUP 

R, FB_ R„ 

(td  = 0)  (tr  = n) 

In  research  on  the  DRE,  the  delay  group  consistently  outperformed 
the  no-delay  group  on  the  R^  retention  measure.  This  held  true  with  such 

diverse  materials  as  French  vocabulary  (Brackbill,  Wagner,  and  Wilson, 

1964;  Markovitz  and  Renner,  1966),  paired-associated  (Sturges,  Sarafino, 
and  Donaldson,  1968),  multiple-choice  items  (Sassenrath  and  Yonge,  1968, 
1969;  Sturges,  1969,  1972),  and  prose  passages  (Moore,  1969;  Surber  and 
Anderson,  1975).  One  factor  common  to  many  of  these  studies  was  the  high 
error  rate  exhibited  at  R^.  This  initial  high  error  rate,  coupled  with 

the  fact  that  the  percentage  of  correct  answers  for  the  no-delay  subject 
changed  very  little  from  R^  to  R^,  lead  Kulhavy  and  Anderson  (1972)  to 

hypothesize  that  the  DRE  occurs  because  the  delay  groups  are  able  to  rectify 
R^  errors  over  the  d^  interval.  Essentially,  the  Kulhavy  and  Anderson 

study,  which  was  later  replicated  by  Surber  and  Anderson  (1975)  shows: 

1.  The  probability  of  a student's  repeating  an  initial  error  on 
a posttest  is  significantly  lower  when  feedback  is  delayed  for  a 24-hour 
period. 


2.  Subjects  who  made  incorrect  responses  and  who  attempted  to 
identify  their  original  response  while  receiving  feedback  were  far  less 
likely  to  remember  errors  following  a delay  interval.  (Of  course,  a delay 
has  the  same  effect  on  those  giving  correct  answers  but  in  this  case, 
feedback  acts  as  additional  practice.) 

3.  Subjects  spend  significantly  more  time  studying  feedback  if 
it  is  not  presented  immediately  after  a difficult  test. 

These  findings  lead  Kulhavy  and  Anderson  to  conclude  that  initial 
error  responses  interfere  proactively  with  acquisition  of  the  correct 
answer  when  the  subject  sees  feedback  immediately  simply  because  the 
item  stems  are  identical  and  the  responses  similar.  However,  this 
interference  is  reduced  when  students  are  given  a chance  to  forget  their 
incorrect  responses  by  providing  a delay  interval  between  response  and 
feedback.  The  set  of  conditions  prevailing  in  the  Kulhavy  and  Anderson 
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study  was  almost  Identical  to  that  encountered  In  list  learning  experiments 
used  to  define  the  proactive  interference  paradigm  (Underwood  and  Freund, 
1968) . 


Learner  Comprehension 


The  DRE  research  clearly  shows  that  feedback  not  only  works  to 
identify  errors  but,  under  appropriate  conditions,  allows  the  subject  to 
correct  them.  Interestingly,  there  are  studies  in  which  even  the  repeated 
administration  of  feedback  falls  to  correct  error  responses  (Elley,  1966; 
Kaess  and  Zeaman,  1960).  However,  such  results  seem  to  occur  only  when  the 
experimental  materials  are  so  unfamiliar  to  the  subjects  that  they  spend 
most  of  their  time  guessing  randomly  at  answers.  In  fact,  Kulhavy  and 
Parsons  (1972)  suggest  that  feedback  will  have  only  minimal  effect  if 
the  learner  is  unable  to  comprehend  the  instruction  or  fit  it  into  some 
existing  information  framework.  In  such  cases,  subjects  are  likely  to 
spend  their  time  trying  to  rotely  associate  the  feedback  with  individual 
items.  It  is  not  surprising  then  that  people  who  learn  only  that  "answer 
A is  correct  for  question  13"  are  likely  to  need  several  repetitions  before 
they  learn  the  proper  paired-associate.  The  comprehension  notion  gains 
even  further  support  from  the  fact  that  most  research  showing  strong  perse- 
veration effects  uses  extremely  convoluted  material,  a condition  providing 
minimum  information  on  which  to  base  initial  responses.  Based  on  this 
reasoning,  it  appears  that  feedback  performs  its  corrective  function  effec- 
tively only  if  mistakes  result  from  faulty  interpretation  rather  than 
lack  of  understanding.  If  learners  fail  to  comprehend  the  material  in 
the  first  place,  feedback  will  have  little  effect  on  performance. 


Feedback  and  Learner  Confidence 


One  feature  which  most  feedback  studies  have  in  common  is  a tendency 
to  measure  performance  only  in  binary  terms;  that  is,  a given  response 
is  considered  either  absolutely  right  or  absolutely  wrong.  Such  an  acute 
dichotomy  makes  little  sense  to  people  who  are  sohpisticated  test  performers. 
It  seems  more  likely  that  in  making  a response  to  a test  item,  most  students 
first  eliminate  obviously  wrong  choices,  and  then  select  from  the  remaining 
alternatives  the  answer  which  seems  most  approparlte.  In  fact,  at  Arizona 
State  trial  data  has  been  acquired  which  indicate  that  students  create 
a hierarchy  of  confidence  in  the  alternatives  and  then  choose  the  most 
probable  right  answer.  Hence,  the  usage  students  make  of  feedback  following 
a response  depends  largely  on  the  interaction  between  their  initial  level 
of  confidence  in  their  response  and  the  accuracy  of  the  answer.  This  con- 
tention allows  one  to  make  predictions  about  learning,  based  on  a know- 
ledge of  how  the  person  perceived  the  item,  and  whether  or  not  he  answered 
it  correctly. 

When  response  confidence  is  high  and  the  correct  answer  is  chosen, 
feedback  probably  receives  only  cursory  attention  since  it  merely  provides 
confirmation.  In  fact,  it  appears  that  the  surer  a student  is  of  his 
response,  the  less  likely  he  is  to  request  feedback  if  it  is  not  readily 
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available  (Melching,  1966).  Hence,  if  a learner  has  high  confidence  in  a 
correct  response,  he  should  spend  a minimum  amount  of  time  studying  feed- 
back and,  in  keeping  with  the  earlier  comprehension  argument,  produce  high 
test  performance  on  such  items. 

When  response  confidence  is  high  but  the  answer  chosen  is  wrong, 
the  situation  is  markedly  different.  Instead  of  briefly  scanning  the 
feedback,  students  are  likely  to  spend  substantial  time  trying  to  locate 
the  source  of  their  mistake.  When  a learner  was  "sure"  that  he  understood 
the  material  but  responded  incorrectly,  he  probably  behaves  much  as  he  would 
in  a class  examination  review;  that  is,  he  becomes  incensed  over  missing 
the  question,  and  pays  a great  deal  of  attention  to  why  the  answer  was 
Incorrect.  Consequently,  feedback  should  play  its  greatest  corrective  role 
with  high  confidence  errors,  simply  because  the  person  spends  more  time 
in  correcting  his  misconception. 

When  response  confidence  is  low,  as  would  be  the  case  when  students 
are  unable  to  understand  the  material,  the  question,  or  both,  they  spend 
their  time  developing  associative  strategies,  rather  than  integrating  new 
information  into  existing  structures.  Under  this  condition,  feedback 
should  have  minimal  effect  regardless  of  whether  or  not  the  response  is 
correct. 

The  set  of  predictions  relating  response  confidence  and  feedback  are 
presented  graphically  in  Figure  3,  which  represents  a right-hand  continua- 
tion of  the  events  shown  in  Figure  2.  To  test  this  model,  a simple  two- 
group  study  was  conducted  in  our  laboratory  in  which  undergraduates  worked 
through  a 30-frame  program  on  the  structure  of  the  human  eye.  Subjects 
in  both  groups  read  each  frame,  answered  a multiple-choice  questions,  and 
rated  their  response  confidence  on  a five-point  scale.  Following  the 
rating,  subjects  in  one  group  received  feedback,  and  those  in  the  second 
group  merely  advanced  to  the  next  frame.  Finally,  all  subjects  were  tested 
immediately  after  the  learning  process  and  again  after  a 1-week  delay. 

With  minor  exceptions,  results  were  as  predicted  by  the  sequence  shown 
in  Figure  3.  High  confidence  responses  followed  by  feedback  were  remembered 
significantly  better  on  the  tests  regardless  of  whether  or  not  the  initial 
response  was  correct.  However,  there  were  no  major  differences  between 
test  scores  of  students  making  low  confidence  responses.  As  shown  in 
Figure  4,  students  who  have  high  response  confidence  and  who  give  the  right 
answer  spend  little  time  studying  feedback,  whereas  students  who  have  high 
response  confidence  and  given  the  wrong  answer  spend  considerable  time. 
Students  making  low  confidence  responses  fall  somewhere  in  between. 

These  results  clearly  show  that  the  interaction  between  learner  response 
confidence  and  feedback  has  a powerful  effect  on  what  the  student  remembers 
from  a lesson.  Thus,  it  appears  that  both  the  form  and  method  of  presenta- 
tion of  feedback  can  be  manipulated  more  effectively  if  they  are  in  some 
way  made  contingent  on  learner  confidence,  particularly  if  the  lesson  para- 
meters are  under  computer  control.  As  a step  in  this  direction,  data  are 
currently  being  collected  on  rated  confidence  across  such  phenomenon  as 
the  DRF.,  and  attempts  are  being  made  to  define  what  aspects  of  the  feedback 
message  are  important  for  error  correction.  This  seems  to  be  a prime 
area  for  future  research. 


MEAN  SECONDS  SPENT  STUDYING  FEEDBACK 


1 2 3 4 5 

CONFIDENCE  IN  PROGRAM  RESPONSE 

Figure  4.  Feedback  study  times  as  a function  of  confidence  ratings. 
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CONCLUS IONS 


Feedback  following  an  instructional  response  appears  to  facilitate 
learning  only  under  certain  specified  conditions.  Feedback  has  little 
facilitative  effect  when  learners  are  able  to  access  the  correct  answer 
prior  to  processing  the  material  and  responding.  In  such  cases,  they  are 
likely  to  simply  copy  responses  and  learn  little  of  the  material  presented. 
Only  when  feedback  is  unavailable  until  after  responding  is  there  likely 
to  be  a facilitative  effect.  The  second  case  where  feedback  is  ineffective 
occurs  when  the  material  to  be  learned  is  very  difficult  and  unfamiliar 
to  the  student.  Here,  learners  tend  to  spend  their  time  guessing  at 
answers,  and  even  if  their  choice  is  correct,  the  amount  of  learning  is 
limited  since  initial  comprehension  was  low. 

When  factors  such  as  copying  and  low  comprehension  are  controlled, 
feedback  appears  to  have  two  primary  effects  on  what  is  learned.  First, 
it  acts  to  confirm  correct  responses,  telling  the  student  how  well  the 
content  is  being  understood.  Second,  feedback  identifies  error  responses 
and  provides  the  means  for  the  learner  to  correct  his  mistakes.  The 
corrective  function  appears  to  be  the  most  important  one,  and  if  there 
is  a choice  to  be  made,  feedback  following  wrong  responses  probably  has 
the  greatest  positive  effect. 

The  material  presented  in  the  final  section  of  the  report  indicates 
that  how  the  student  perceives  his  understanding  of  the  material  has  a 
great  deal  to  do  with  what  benefit  he  gets  from  feedback.  In  cases 
where  the  lesson  is  well  understood  and  the  learner  is  confident  that 
his  initial  response  is  correct,  he  will  tend  to  get  the  information 
correct  on  a later  test.  Interestingly,  this  effect  holds  regardless 
of  whether  the  initial  response  was  correct,  since  students  making  high 
confidence  wrong  responses  make  maximum  use  of  the  corrective  function 
of  feedback.  These  data  suggest  that  how  a student  evaluates  a response 
has  much  to  do  with  the  use  to  which  he  will  be  able  to  put  feedback.  It 
appears  that  measures  of  the  student  response  expectations  may  be  an 
effective  way  in  which  to  provide  data  for  use  by  a computer-controlled 
system  in  evaluating  what  type  of  information  should  follow  the  response. 


2.  Minimize  the  opportunity  for  students  to  obtain  feedback  prior 
to  responding. 

3.  Provide  feedback  as  often  as  possible  during  the  course  of  the 
lesson. 


It  is  further  recommended  that  future  research  in  this  area  be  directed 
toward  the  relationship  between  learner  confidence  and  feedback. 
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